SELECTIVE PROCESSING OF DATA EMBEDDED 

IN A MULTIMEDIA FILE 

CROSS REFERENCES TO RELATED APPLICATIONS 

This application is a continuation-in-part of U.S. Patent Application Serial 
No. 09/790,032 filed February 20, 2001, entitled "Linking Internet Documents with 
Compressed Audio Files" which claimed the benefit of U.S. Provisional Application 
60/183,765 filed February 18, 2000, 

FIELD OF THE INVENTION 

The present invention generally relates to embedding data into a media file, and, 
more particularly, where time restriction information is included. 

BACKGROUND OF THE INVENTION 

The process of compressing an audio source, such as voice or music, and storing 
it digitally in a file is conventionally known. A user connected to the Internet can 
selectively download an encoded compressed file, such as a file in the MP3 format, run 
software to decode the file and thereby Hsten to the audio source. The MP3 format is 
well known in the art and refers to Layer 3 audio files of the Motion Picture Experts 
Group (MPEG) intemational standard for representation, compression, and 
decompression of motion pictures and associated audio on digital media. 
Conventionally, a decoder is employed to decode the encoded digital audio file. 

A drawback of conventional methods is that compressed audio files do not 
include information and links to Internet documents that can be viewed during audio 
playback. Conventional methods require the Ustener of the decoded audio file to 
manually make any such Hnks to Intemet documents. 

Methods are known for embedding data at a predetermined point in a media file, 
which may include, but is not hmited to, an audio file. Such methods provide a means of 
permanently embedding data in a media file, extracting and displaying or further 
processing the embedded data on a suitable decoding system. For example, an Intemet 
Uniform Resource Locator (URL) link to "www.intervideo.com/promotion.htm" can be 
embedded in a music file. Using these methods, whenever the music file is decoded and 
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played, the corresponding decoder application can enable the web page corresponding to 
the embedded link to display a promotional message at a point in time during playback of 
the music file. For another example, a promotional message can be embedded in a music 
file such that when the music is decoded and played, the corresponding decoder 
application enables display of the message at a point in time in the middle of a particular 
song being played back. A drawback with these methods is that the embedded data 
permanently remains in the file and the decoder will always try to display the 
information. For the first above example, if the web page "promotion.htm" is removed 
fi:-om the web site "www.intervideo.com", the conventional decoder tries to load that page 
at a point in time during playback and will always display an error or a default message 
to indicate that the decoder could not find or display the information. For a fiarther 
example, if the following text "Buy this CD now to get 50% discount" is embedded in a 
song to promote a limited duration discount offer, the text will be displayed every time 
the song is played back, even after the discount offer has expired. 

One way to address the above drawbacks is to specify the offer duration in the 
message, for example, by changing the text to read "Buy this CD now to get 50% 
discount, offer expires on June 25 , 2001 at midnight". Even though this solves the 
ambiguity of offer duration, the message will still be displayed permanently at a point in 
time during playback even years after the offer has expired. Another approach is to 
remove the embedded data after a certain amount of time, but this entails altering the 
media file, which requires additional processing and may not be desirable or feasible. A 
similar problem exists when the intention is to display the information only after a certain 
date. 

What is needed is a method whereby information embedded in a media file can be 
extracted only during a predetermined time duration. 

SUMMARY OF THE INVENTION 

The aforementioned drawbacks of known methods associated with permanently 
embedding data in a media file are overcome by the present invention. The present 
invention is directed to a method for encoding of media files to embed a data segment 
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and time restriction information into the media file in order to restrict the time period 
when decoding and processing of the embedded data segment can be performed. The 
media file is capable of being decoded for playback as a media stream. The inventive 
method also includes a method for decoding the encoded media file such that the data 
segment is reproduced for playback as part of the media stream only during a specified 
time period as compared to the current time of day and/or date when the media stream is 
being played back. 

Li the preferred embodiment, to indicate a time restriction for the decoding of the 
data embedded in a media file, a method is provided to embed special time tags along 
with the embedded data segment to indicate a predetermined time when the data segment 
can be decoded and a predetermined stop time, after which time the data can no longer be 
extracted. The method for decoding includes reproducing the data segment at a specified 
point in the decoded media stream for playback only during the time period specified by 
the time tags. The reproducing of the embedded data segment preferably includes 
processing the embedded data segment to enable the embedded data segment or Mtemet 
linked web page or document, however, any suitable processing may be used. 

Li accordance with another aspect of the invention, a method is provided for 
decoding the encoded media file such that the data segment is reproduced for playback as 
part of the media stream only during a time period specified by one or more time tags 
embedded along with the data segment. Each time tag is compared to the current time of 
day and/or date when the media stream is being played back to determine if the data 
segment is to be reproduced for playback. 

In one embodiment, the data segment is an address of an Internet document and is 
embedded along with the corresponding time restriction information into a media file. 
For this embodiment, during decoding the embedded Internet address is reproduced and 
processed for playback such that the web page or document corresponding to the Internet 
address is only displayed during a time period specified by the time restriction 
information as compared to the current time of day and/or date when the media stream is 
being played back. 

For the present invention the media files may include, but are not limited to, audio 
files. Any suitable media files may be used, including MPS files. The present invention 
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is also not limited to the use of files, a suitable stream, not stored as a file, may be used. 
Preferably, the media file is compressed prior to the embedding of data and time 
restriction information, however, the present invention is not limited to use for 
embedding data and time restriction information into compressed media files, any 
suitable media file may be used. 

In an encoder for encoding a media file so that it is capable of being decoded for 
playback as a media stream, the present invention is a method for encoding data and time 
restriction information with the media file such that when the encoded media file is 
decoded for playback as the media stream, the data is reproduced in the media stream 
only during a time period predetermined by the time restriction information, comprising 
the steps of: receiving a media file; receiving at least one data file comprising at least one 
data segment; receiving time restriction information for each data segment that specifies 
the time period when the data segment is to be reproduced as part of the decoded media 
stream; and embedding as part of the encoded media file each data segment and 
corresponding time restriction information such that the data segment is reproduced by a 
decoder during the specified time period. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates the process steps of an encoder method or algorithm according 
to an exemplary embodiment of the present invention; 

Figure 2 is a flow chart illustrating the process steps of an decoder method or 
algorithm according to an exemplary embodiment of the present invention; 

Figure 3 is a flow chart illustrating an exemplary embodiment of the process steps 
of the encoder method that enables selective presentation time tagging according to a 
preferred embodiment the present invention; and 

Figure 4 is a flow chart flow chart illustrating an exemplary embodiment of the 
process steps of the decoder method that enables selective presentation time tagging 
according to a preferred embodiment of the present invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

The present invention is directed to a method for encoding of media files to 
embed a data segment and time restriction information into the media file in order to 
restrict the time period when decoding and processing of the embedded data segment can 
be performed. In accordance with the present invention, a method is provided for 
specifying the time period determined by the time restriction information. The time 
restriction information comprises embedding with the corresponding data segment only a 
start time tag, embedding only a stop time tag, or embedding both a start time tag and a 
stop time tag. A start time tag specifies when the time period starts for reproducing the 
data segment. A stop time tag specifies when the time period for reproducing the data 
segment is to end. The data segment is no longer reproduced (e.g. including display to a 
user) after the stop time. The method of the present invention provides that both a start 
time tag and a stop time tag may be encoded along with the data segment in order to 
specify when the time period for reproducing the data segment is to begin and when the 
time period ends. 

The "time" indication for the tags is not restricted to "time of day" but preferably 
indicates a date in a suitable date format. The present invention also provides for time 
tags that are a combination of date and time-of-day. For example, start time and stop 
time tags could be specified such that the start of the time period would be in a format of 
hours minutes: seconds and the stop time would be a different hours:minutes:seconds such 
that the time period would be independent of month, day and year. Another example is a 
start time tag and stop time tag that each includes month, day, and year information such 
as where the time period starts at one specific date and ends at another specific date. For 
a further example, the start time tag and stop time tag both include hours:minutes:seconds 
and month-day-year restrictions. The time tags are not limited to the above examples, 
any suitable set of time period restrictions can be used. 

The data segment and media file are not Umited to files, suitable data streams may 
also be used. The media file is preferably compressed. The present invention is not 
limited to use with compressed media files, however, any suitable uncompressed media 
file may be used. 

For the preferred embodiment of the inventive method of the present invention the 
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method of encoding is in a encoder which is for encoding a media file so that it is capable 
of being decoded for playback as a media stream. Figure 1 illustrates the process steps of 
an encoder method or algorithm 10 according to an exemplary embodiment of the present 
invention. Li one exemplary aspect, to link documents (i.e. Litemet documents) with 
compressed audio jSles such as ".MP3" files, the addresses of the Intemet documents 
along with the corresponding "point in time" information are embedded into the 
compressed files. The point in time information specifies the point within the media 
stream when the embedded data Litemet document, or other displayable information, 
should be reproduced when the media file is played back by a decoder, hi the case of 
compressed audio files, when the encoded file is played back in a decoder which is aware 
(i.e. can detect) of such an encoding scheme, it extracts the embedded information while 
decompressing the audio information and uses the embedded information to reference the 
documents on the Litemet. 

An example of the data that can be embedded in a media file according to this 
aspect of the present invention is as follows: 

{ 

1.0: www.intervideo.com 

10.2: www.intervideo.com/mymusic/infol .htm 

} 

The number on the left indicates the timing information. For use with compressed 
audio files, an audio compressor unit of the encoder uses these timing values to embed 
the corresponding information at the specified point in time. The decoder extracts this 
information and executes a proper action. The embedded information may also represent 
displayable information that can be displayed. The information can be a web page link 
such that the decoder can open the corresponding web page or web command at the 
specified times. Li another aspect, the encoder embeds the point in time information 
along with the corresponding data. The method can be performed in various systems 
including a computer system or other device that includes a central processing unit and a 
display. Preferably, the system has the capability to connect to the Litemet. 

For the method or algorithm 10 in Figure 1, at Step 20 the encoding process 
begins and during this step the decoder gets a piece of information (e.g. a text character 
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"c") to embed. The information is shown as text character information, however, aay 
information the decoder can display or otherwise process can be embedded. In Step 30 
the encoder reads a block of audio data samples, shown as x_i. For the example in Fig. 1, 
the data samples can be frequency domain MPS compressed audio samples. Step 30 also 
shows, as an example, the reading of a frame. It is well known in the art that an MP3 
bitstream comprises frames of compressed data. It is also known there may be more than 
one audio channel, e.g. stereo, with each channel storing independent samples. The 
algorithm shown in Fig. 1 can also be applied to embed different information in each 
channel. At Step 40, the encoder determines whether the maximum value of the audio 
sample in the block exceeds a threshold value. The threshold value has been determined 
to be a value below which the embedding of data would unacceptably degrade the audio 
quahty for a user during playback. If the threshold value has not been exceeded by the 
maximum value, then the method jumps back to Step 30. If the threshold has been 
exceeded, then in Step 50 the encoder modifies the least significant bits (LSB's) of a 
subset of said block, wherein a digital representation of the information (shown as 
exemplary text character "c" in Figure 1) is encoded. Preferably, the subset is a block of 
16 samples, as shown in Step 50 {x_k through x_k+15}; where k=mod(i_max,16). In 
Step 60, a determination is made as to whether the maximum of the modified block 
samples (shown as x_i_max) exceeds the threshold, shown as "thr". If the threshold is 
exceeded, embedding has been successful and there is a branch to Step 20. If the 
threshold is not exceeded then two is added to the digital value of the modified sample to 
exceed said threshold. The value two is chosen because it is the smallest value that can 
be added to a number without changing the least significant bit of the binary 
representation. The current invention is not limited to this value, however. Then the 
process returns to Step 20 for the encoding of the next character. 

For the preferred embodiment of the inventive method of the present invention the 
method of decoding is in a decoder which is for decoding an encoded media file for 
playback as a media stream. Figure 2 is a flow chart illustrating the process steps of a 
decoder method or algorithm 110 according to an exemplary embodiment of the present 
invention. According to this method, what is decoded is information (shown as text 
character "c" for the exemplary algorithm) that was previously encoded as specified in 
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the encoding method shown in Figure 1. As seen in Figure 2, in Step 120 the decoder 
reads a block of encoded audio data samples. At Step 130, a determination is made as to 
whether the maximum value of the encoded audio sample in the block exceeds a 
threshold value. If it is above the threshold, this indicates that no text characters were 
embedded. If the threshold is not exceeded, then the process returns to Step 120. If the 
threshold is exceeded, then Step 140 is performed wherein the decoder reads the least 
significant bits (LSB's) of a subset of the encoded block, wherein a digital data code is 
decoded. In Step 150, the decoder determines whether the decoded digital data code 
represents vahd information (shown as "c" for the exemplary algorithm) wherein the 
decoded information is found in an expected set. An ASCII character set is an example 
of an expected set. An expected set can be chosen in a manner suitable for the desired 
appUcation. It is also possible that some applications do not require limiting the 
embedded information to an expected set. The present invention is not limited to ASCII 
characters, for example. If the decoded information is not in the expected set, then the 
process proceeds back to Step 120. If the decoded information is in the expected set, then 
Step 160 is performed wherein the decoded valid information (e.g. exemplary character 
"c" in Step 160) is added as a new character (e.g. to an existing string), and the process 
then proceeds back to Step 120. 

The drawbacks of known methods associated with permanently embedding the 
data in a media file are overcome by the present invention. In the preferred embodiment, 
a method is provided to encode time restriction information along with each 
corresponding data segment embedded in a media file in order to specify a time period 
such that the data segment is reproduced by a decoder at a specified point in the media 
file only during the time period determined by the time restriction information. The data 
segment is preferably part of a data file. The time restriction information is comprised of 
one or more time tags in order to specify the proper time period to reproduce the data 
segment embedded in the media file. The reproducing of the data segment preferably 
includes subsequent processing for enabling the data to be displayed to a user. The 
present invention is not limited to such processing however, any suitable processing can 
be performed. 

For the preferred embodiment of the present invention, during encoding of a 
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media file to embed data, two checks are done before each data segment is embedded. 
According to one preferred encoding method, a first check is made to determine if the 
data segment is to have a "start time" associated with it. A farther check is made to 
determine if the data segment is to have a "stop time" associated with it. These checks 
could be done by either interactively prompting the file creator or through pre- 
programming a special script file. If the data segment is to be reproduced for a limited 
time period, then during the encoding process one or more corresponding time tags are 
embedded along with the data for that segment. Preferably, the time tags are encoded 
such that the tags include a unique identifier to specify the tag type (start or stop) and a 
"time" field that specifies the time to start or stop presenting the data. As shown in the 
above examples, the "time" field can comprise both date and time information. The 
preferred embodiment of the present invention will now be described in detail with 
reference to Figures 3 and 4. 

Fig. 3 is a flow chart illustrating exemplary process steps of the encoder algorithm 
200 for the encoding of media files to embed a data segment and time restriction 
information into the media file in order to restrict the time period when decoding and 
processing of the embedded data segment can be performed. The encoding method 
enables selective presentation time tagging according to a preferred embodiment of the 
present invention, hi Step 210 of Figure 3, the encoding algorithm begins encoding of a 
media file. In step 220, a determination is made as to whether there is a data segment to 
be embedded in the media file. If there is no data segment to embed, the encoding 
algorithm proceeds to finish at Step 230. If a data segment to be embedded does exist, 
then Step 240 is executed. The method in Figure 3 then determines whether to embed 
time restriction information that specifies the time period when the data segment is to be 
reproduced as part of a decoded media stream. 

At Step 240, a check is made as to whether the data segment has to be reproduced 
(e.g. presented) only after a certain "date" where "presented" refers to enabling the data 
to be displayed to a user for the preferred embodiment. For the preferred embodiment in 
Fig. 3, a certain "date" is shown in the exemplary embodiment in the algorithm. For the 
present invention the "date" is not limited to a calendar date, however. Any suitable 
time-of-day or combination of time-of-day and date can be used for the time restriction 
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information. If in Step 240 it is determined that the data segment is to be presented after 
a certain date, then the encoding algorithm proceeds to Step 250. hi Step 250, for the 
media file a corresponding start_time tag is embedded with the data segment as part of 
the media file and then Step 260 is executed. The start_time tag specifies when the time 
period starts for reproducing the data segment. If in Step 240 is it determined that the 
data segment is not limited to only being presented after a certain date, then the algorithm 
Step 260 is executed. 

In Step 260, a determination is made as to whether the data segment should not be 
presented after a certain date. If the data segment should not be presented after a certain 
date (e.g. the time period ends at a certain date), then Step 270 is executed, hi Step 270 a 
stop_time tag is embedded in the media file, and the algorithm proceeds to Step 280. A 
stop time tag specifies when the time period for reproducing the data segment is to end. 
As seen in Figure 3, the method provides that both a startjime tag and a stopjime tag 
may be encoded along with the data segment in order to specify when the time period for 
reproducing the data segment is to begin and when the time period ends. If in Step 260 it 
is determined that there is not a restriction specifying that the data segment should not be 
presented after a certain date, then Step 280 is executed. In Step 280, the data segment is 
embedded as part of the media file, and the algorithm proceeds back to Step 220 to check 
for more data segments. 

The inventive method also includes a method for decoding the data segment as 
the encoded media file is reproduced as a media stream only during a specified time 
period as compared to the current time of day and/or date when the media stream is being 
played back. Fig. 4 is a flow chart illustrating the process steps of a preferred 
embodiment of the method or algorithm for decoding for the present inventive method 
that enables selective presentation time tagging. The decoder algorithm 300 begins at 
Step 310 for processing of the encoded media file. In Step 320 a determination is made 
as to whether an embedded data segment exists in the encoded media file. If there is not 
an embedded data segment, then the method proceeds to Step 330 to finish display of 
embedded information. If in Step 320 it is determined that an embedded data segment 
does exist in the encoded media file, then Step 340 is executed. In Step 340 a 
determination is made as to whether a startjime tag exists in the data segment such that 
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the data segment is only to be reproduced for a time period starting at the specified start 
time of the start__time tag. If the start_time tag exists in the data segment, Step 350 is 
executed. Li step 350, the decoding algorithm obtains the current time and determines 
whether a current time is before the specified start_time in the start_time tag. If the 
current time is before the start time, the method loops back to Step 320. If the current 
time is not before the start time, then Step 360 is executed. Also, if at step 340 a 
start_time tag does not exist then Step 360 is executed. 

As shown in Fig. 4, at Step 360 a determination is made as to whether a stopjime 
tag_exists in the data segment such that the data segment is only to be reproduced for a 
time period ending at the specified stop time of the stop_time tag. If a stop_time tag 
exists, then the algorithm proceeds to Step 370. In Step 370, a determination is made as 
to whether the current time is after the stop_time tag. If the current time is after the 
stop_time tag, then the algorithm loops back to Step 320. If the current time is not after 
the stop time in the stop_time tag then Step 380 is executed. If it is determined in Step 
360 that a stop_time tag does not exist, then Step 380 is executed. In Step 380 the data 
segment is reproduced (e.g. displayed in the exemplary embodiment in Fig. 4) and then 
the method loops back to execute Step 320 for the next data segment. Preferably, the 
reproducing of the data segment includes subsequent processing for enabling the data to 
be displayed to a user. However, any suitable processing may be performed. 

For the exemplary embodiment shown in Figure 4, the "time" is referred to in the 
algorithm, however, for the invention the "time" is not hmited to a clock time, a suitable 
time, day or time and date representation can also be used. The current "time" shown in 
Figure 4 may include time of day and date information obtained by the decoding method 
through suitable means, the details of which are well known in the art and beyond the 
scope of this description. 

In another embodiment, one or more time tags are embedded with the data 
segment in the encoded media file but the current time is not available for comparison 
purposes when the media stream is being played back during decoding. When the current 
time is not available for comparison; the decision as to whether or not the embedded data 
segment is reproduced can be made by a user of the inventive method. AUematively, 
whether the data segment is to be reproduced or not reproduced when the current time is 
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not available can be predetermined (e.g. by the data segment or media file creator). 

While the present invention has been particularly described with respect to the 
illustrated embodiments, it will be appreciated that various alterations, modifications and 
adaptations may be made based on the present disclosure, and are intended to be within 
the scope of the present invention. While the invention has been described in connection 
with what is presently considered to be the most practical and preferred embodiments, it 
is to be understood that the present invention is not limited to the disclosed embodiments 
but, on the contrary, is intended to cover various modifications and equivalent 
arrangements included within the scope of the appended claims. 
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